Stochastic ADMM for Nonsmooth Optimization

Alternating Direction Method of Multipliers (ADMM) gained lost of attention due to LargeScale Machine Learning demands. • Classic (70’s) and flexible, Survey paper: (Boyd 2009) • Applications: compressed sensing (Yang & Zhang, 2011), image restoration (Goldstein & Osher, 2009), video processing and matrix completion (Goldfarb et al., 2010) • Recent variations: Linearized (Goldfarb et al., 2010; Zhang et al., 2011; Yang & Yuan, 2012), Accelerated (Goldfarb et al., 2010) and Online (Wang & Banerjee, 2012) ADMM • Global convergence proved in 80s ((Gabay, 1983; Eckstein & Bertsekas, 1992)) • Recent progress on rate of convergence: O(1/T ) for convex functions (He’11) • We propose a linearized stochastic ADMM algorithm; applies to a more general class of convex and nonsmooth objective functions, beyond the smooth and separable least squares loss used in lasso.